AN EXCURSION-THEORETIC APPROACH TO STABILITY OF DISCRETE-TIME 
STOCHASTIC HYBRID SYSTEMS 
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Abstract. We address stability of a class of Markovian discrete-time stochastic hybrid 
systems. This class of systems is characterized by the state-space of the system being 
partitioned into a safe or target set and its exterior, and the dynamics of the system 
being different in each domain. We give conditions for Lj -boundedness of Lyapunov 
functions based on certain negative drift conditions outside the target set, together 
with some more minor assumptions. We then apply our results to a wide class of ran- 
domly switched systems (or iterated function systems), for which we give conditions 
for global asymptotic stability almost surely and in Lj . The systems need not be time- 
homogeneous, and our results apply to certain systems for which functional-analytic 
or martingale-based estimates are difficult or impossible to get. 



1. Introduction 

Increasing complexity of engineering systems in the modern world has led to the 
hybrid systems paradigm in systems and control theory [vSOO, Lib03] . A hybrid system 
consists of a number of domains in the state-space and a dynamical law corresponding 
to each domain; thus, at any instant of time the dynamics of the system depends 
on the domain that its state is in. One would then restrict attention to behavior of 
the system in individual domains, which is typically a simpler problem. However, 
understanding how the dynamics in the individual domains interact among each other 
is necessary in order to ensure smooth operation of the overall system. This article 
is a step towards understanding the behavior of (possibly non-Markovian) stochastic 
hybrid systems which undergo excursions into different domains infinitely often. Here 
we consider the simplest and perhaps the most important hybrid system, consisting of 
a compact target or safe set and its exterior, with different dynamics inside and outside 
the safe set. Our objective is to introduce a new method of analysis of systems that are 
outside the safe set infinitely often in course of their evolution. The analysis carried 
out here provides a basis for controller synthesis of systems with control inputs — it 
gives clear indications about the type of controllers to be designed in order to ensure 
certain natural and basic stability properties in closed loop. 

Let us look at two interesting and practically important examples of hybrid systems 
with two domains — a compact safe set and its exterior, with different dynamics in each. 
The first concerns optimal control of a Markov process with state constraints. Markov 
control processes have been extensively studied; we refer the reader to the excellent 
monographs and surveys [BS78, Bor91, HLL96, HLL99] for further information, ap- 
plications and references. For our purposes here, consider the canonical example of a 
hnear controlled system perturbed by additive Gaussian noise and having probabilistic 
constraints on the states. A hybrid structure of the controlled system naturally presents 
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itself in the following fashion. Except in the most trivial of cases, computing the con- 
strained optimal control over an infinite horizon is impossible, and one resorts to a 
a rolling-horizon controller (Rolling-horizon controllers are considerably popular, for 
basic definitions, comparisons and references see e.g., [MacOl] in the deterministic 
context, and [CHL09] and the references therein in the stochastic context.) Computa- 
tional overheads restrict the size of the window in the rolling-horizon controller, and 
determine the maximal (typically bounded) region — called the safe set — in which this 
controller can be active. No matter how good the resulting controller is, the addi- 
tive nature of the Gaussian noise ensures that the states are subjected to excursions 
away from the safe set infinitely often almost surely. Once outside the safe set, the 
rolling-horizon controller is switched off and a recovery strategy is activated, whose 
task is to bring the states back to the safe set quickly and efficiently. This problem is 
of great practical interest and a subject of current research, see e.g., [CHL09] and the 
references in them for possible strategies inside the safe set, and [CCCL08] for one 
possible recovery strategy. Evidently, stability of this hybrid system depends largely 
on the recovery strategy, since as long as the states stay inside the safe set, they are 
bounded. However, traditional methods of stability analysis do not work well precisely 
because of the unlimited number of excursions. Theorem (2.2) of this article addresses 
this issue, and provides a method of ensuring strong boundedness and stability prop- 
erties of the hybrid system. Intuitively it says that under the recovery strategy there 
exists a well-behaved supermartingale until the states hit the safe set, then the system 
state is bounded in expectation uniformly over time. A complete picture of stability 
and ergodic properties of a general controlled hybrid system is beyond the scope of 
the present article, and will be reported elsewhere. We refer the reader to [CFM05, 
Chapter 3] for earlier work pertaining to stability of a class of hybrid systems, and 
to [MT93] for stability of general discrete-time Markov processes. 

The second example is one that we shall pursue further in this article, namely, a class 
of discrete-time Markov processes called iterated function systems [BDEG88, LM94] 
(iFs). They are widely applied, for instance, in the construction of fractals [LM94], in 
studies on the process of generation of red blood corpuscles [LM02, LS04], in statisti- 
cal physics [Kif86], and simulation of important stochastic processes [Wer05]. Of late 
they are being employed in key problems of physical chemistry and computational biol- 
ogy, namely, the behavior of the chemical master equation [Wil06, Chapter 6] (CME), 
which governs the continuous-time stochastic (Markovian) reaction-kinetics at very 
low concentrations (of the order of tens of molecules). Invariant distributions, certain 
finite-time properties, and robustness properties with respect to disturbances of the 
underlying Markov process are of interest in modeling and analysis of unicellular or- 
ganisms. It is well-known that the CME is analytically intractable (see [JH07, ACK08] 
for special cases), but the invariant distribution of the Markov process can be recov- 
ered from simulation of the embedded Markov chain in a computationally efficient 
way [MA08]. This embedded chain is an ifs taking values in a nonnegative integer 
lattice. From a biological perspective, good health of a cell corresponds to the ifs 
evolving in a safe region on an average, despite moderate disturbances to the numbers 
of molecules involved in the key reactions. However, in most cases compact invari- 
ant sets do not exist. It is therefore of interest to find conditions under which, even 
though there are excursions of the states away from a safe set infinitely often, the ifs 
is stochastically bounded, or some strong stability properties hold. Theorem (2.2) of 
this article leads to results (in §3) which address this issue. 

This article unfolds as follows. §2 contains our main results — Theorem (2.2) and (2.9), 
which provide conditions under which a Lyapunov function of the states is Lj -bounded. 
We establish this Lj -boundedness under the assumptions that a certain derived process 
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is a supermartingale outside a compact set, and some more minor conditions.^ (The 
supermartingale condition alone is not enough, as pointed out in [PR99], where the 
authors establish variants of our results for scalar, possibly non-Markovian processes 
having increments with bounded p-th moments for p > 2.) For our results to hold, the 
underlying process need not be time-homogeneous or Markovian. To wit, in §2.2 we 
define a class of hybrid processes that switch between two Markov processes depend- 
ing on whether they inside or outside a fixed set in the state-space, and demonstrate 
that although the resulting process may be non-Markovian, our results continue to 
hold. Connections to optimal stopping problems are drawn in §2.3, which gives a sys- 
tematic procedure for verifying our assumptions. In §2.4 we apply the techniques our 
techniques to a class of sampled diffusion processes. In addition to the cases consid- 
ered here, the results in §2 will be of interest in queueing theory, along the lines of the 
works [HLR96, BKR+Ol]. §3 contains some applications of the results in §2 to stability 
and robustness of ifs. The classical weak stability questions concerning the existence 
and uniqueness of invariant measures of ifs, addressed in e.g., [DF99, JTOl, Sza06], 
revolve around average contractivity hypotheses of the constituent maps and conti- 
nuity of the probabilities. In §3.1 we look at stronger stability properties of the ifs, 
namely, global asymptotic stability almost surely and in expectation, for which we give 
sufficient conditions. There are no assumptions of global contractivity or memoryless 
choice of the maps at each iterate; we just require a condition resembling average 
contractivity in terms of Lyapunov functions with a suitable coupling condition with 
the Markovian transition probabilities. We mention that although some of the assump- 
tions in [JTOl] resemble ours, the conditions needed to establish existence of invariant 
measures in [JTOl] are stronger than what we employ; see §3.1 for a detailed com- 
parison. We also demonstrate in §3.2 that under mild assumptions, iterated function 
systems possess strong stability and robustness properties with respect to bounded dis- 
turbances. In this subsection the exogenous bounded disturbance is not modeled as a 
random process. 

Notations. Let N := {1, 2, . . .}, Nq := {0, 1, 2, . . .}, and Rj,o '■= [0,oo[. We let |H| denote 
the standard Euclidean norm on M'' . We let denote the closed Euclidean ball around 
0, i.e., Bf. := {y e K''] ||j|| ^ r}. For a vector v e R'' let v'^ denote its transpose, and 
II V lip denote Vv^Pv for a d x d real matrix P. The maximum and minimum of two real 
numbers a and b is denoted by a V b and a A fe, respectively. 

2. General results 

Before we get into hybrid systems, it will be simpler to follow the arguments if we 
start by considering a discrete-time Markov chain. 

2.1. Obtaining Ij Bound using Excursions. Let X := (Xj^gp^^ be a discrete time 
Markov chain with a state space 5^ . We denote the transition kernel of this chain by 
P, i.e., for every x e 5^, the probability measure P^i-) '■= P(^,0 determines the law 
of Xf+i, conditioned onX^ = x. At this point we only assume the state space 5^ to be 
any Polish space. 

(2.1). Assumption. There exists a nonnegative function : Ng x 5^ — > R~^q satisfying 
the following. 

(i) There exists a subset K cz y such that the process (FJtgNo defined by = 
c^(t,Xt) is a supermartingale under P^^ , for every Xq& y \ K until the first time 



It also seems conceivably possible that relaxed Foster-Lyapunov inequalities as in [DFMS04, Condition 
D(i^, V, C), p. 1356] arising in the context of subgeometric convergence to a stationary distribution can be 
employed in the construction of the aforementioned supermartingale; this constitutes future work. 
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hits K. To wit, if = Xq e 5^ \ K and we define 
= inf{t > |Xt eK}, 

then the process (^£AT^)teNt, ^ supermartingale under P^^. 

(ii) There exists a nonnegative measurable real-valued function V : 5^ — > R and a 
positive sequence (0(t))t€No such that 

(pit,x) ^ VM/9[t) for all (t,x) e Nq x 5^, 

andC:=l],gj,^0(t)<oo. 

(iii) 5 := sup^^^ V{x) < oo. 

Our objective is to prove under the above condition (and another minor assump- 
tions) that there exists a bound on sup^ E^^ depending on Xq. 

(2.2) . Theorem. Consider the setup in Assumption (2.1), and assume that 

(2.3) /3 := sup E [ipiO,Xi)l^x,ey\K] I = ^o] < oo- 

Let Y := sup^g^j^ d[t)- Then we have 

sup E,^ ^ C/3 + 5 + rviO,Xol 

teNo 

In the rest of this section we prove the above theorem. Fix a time t e Nq, and define 
two random times 

gt := sup{s e No I 5 ^ t, e K} and := inf{s e Nq | s ^ t, e K}. 

We follow the standard custom of defining supremum over empty sets to be — oo, and 
the infimum over empty sets to be -l-oo. 

Note that is not a stopping time with respect to the natural filtration generated 
by the process X, although is. The random interval [g^, is a singleton if and only 
if Xf e Otherwise, we say thatX^ is within an excursion outside K. 

Now we have the following decomposition: 

t 

(2.4) E,„ [V(X,)] = E,„ [V(X,)l;,^=_<,j] + Ex„ [V(X,)lj,^=,j] . 

5=0 

Our first objective is to bound each of the expectations E^^ [^(^t)l{g,=5}] ■ 

Before we move on, let us first prove a Lemma which follows readily from Assump- 
tion 1. 

(2.5) . Lemma. LerXo =Xo&5^\K. Then 

(2.6) E,^ [V(X,)l;,^>,j] ^ (^(O,xo)0(s) fors e No, 
where (0(t))tgpj„ defined in Assumption (2.1). 

Proof. This is a straightforward application of Optional Sampling Theorem (OST) for 
discrete-time supermartingales. Applying OST for the bounded stopping time s A to 
the supermartingale (y(t,Xt)) ^.^j^^, in view of ip ^ 0, we have 

¥>(0,Xo)^ E^J(^(sATj^,X,^^^)] > E^j¥>(5,Xjlj^^>,j]. 

Now, by condition (i) in Assumption (2.1), we can write f{,s,x) ^ V[x)/9i^s). Thus, 
substituting back, one has 

(^(0,Xo)>E,„ [V(Xjlj,^>,j]/0(s). 

Since (0(t))teNo positive, we arrive at (2.6). □ 

We are ready for the proof of Theorem (2.2). 
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Proof of Theorem (2.2). Let us consider three separate cases: 



Case 1. (—00 < < t). In this case can take values {0, 1, 2, . . . , f — 1}. Now, if 
5 e {0,1,2,..., t- 1}, then 

Ex„ [^(^t)l{g,=5}] = ^Xo[^(.Xt)^{X,€K}'^{XifK,i=s+l,...,t]~\ 



P(xo,dx) 



P(x,dj)E^[V(X,_,_i)l|.^>t-.-i}], 



and by (2.5) it follows that the right-hand side is at most 



P(xo,dx) 



Pix,dy)ip[0,y)eit-s-n 



Thus, one has 



P(xo,dx) 



(2.7) 



K 



P(x,dy)(^(0,y) 



^ 0(t - s - 1) sup E, [^(.0,X,n{x,e.9-\K}] 

x€K 

= 0(f-s- 1)^. 



Case 2. (g^ = t). This is easy, since implies V[X^) ^ 5. Thus 

E,^ _„] ^ 5P,^(X, e K) ^ 5. 

Case 3. (gf = — oo). This is the case when the chain started from outside K and has 
not yet hit K, and therefore, 

E,JV(Xjl,g^_^,] =E,JV(Xjl{,^>,j] ^^(O,xo)0(t). 

Combining all three cases above, we get the bound: 

t-i 

(2.8) E.jn^J] ^20(f-s-l)^ + 5 + (^(O,Xo)0(t). 

s=0 

Maximizing the right-hand side of (2.8) over t, we arrive at 

oo 

sup E,„ ^pY, 0(5) + 5 + ^iO,Xo) sup 0(t), 



which is the bound stated in the theorem. 



□ 



Often it will turn out that (/^(f, x) is a function ip^t, V(x)) as in the case of the clas- 
sical Foster-Lyapunov type supermartingales [MT93]. In that case i/j(f,x) = e°'^V(x), 
for some positive a. Thus i^(t, ■) is a linear function of V(x) for each fixed t, with 
6[t) = e~"', which shows that the sequence (0(f))t€No summable. See also [FK04] 
and the references therein for more general Foster-Lyapunov type conditions. For ex- 
amples which are not linear see §2.4. 

2.2. A Class of Hybrid Processes. The preceding analysis can be extended for pro- 
cesses which switch their behavior depending on whether the current value is within 
K or not. They constitute a particularly useful class of controlled processes in which 
a controller attempts to drive the system into a target or safe set K cz 5^ whenever 
the system gets out of K due to its inherent randomness. Below we give a rigorous 
construction of such a process. 
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A process X that is (7, Z)-hybrid with respect to K. Consider a pair of Markov chains 
(7, Z) where 7 is a time-homogeneous Markov chain, and Z is a (possibly) time inho- 
mogeneous Markov chain. We construct a hybrid discrete-time stochastic process X by 
the following recipe: 

Firstly, let the state space for the process be 5^^" along with the natural filtration 

<^0 — ^1 — ^2 — * • • 

generated by the coordinate maps. 

Secondly, we define the sequence of stopping times CTq := Tq := — oo and ^ o-^ ^ 
T2 ^ C72 ^ • • • by 

T; := inf{ t > cTj.i I e and 
cr, := inf{t >T^\X,^K} 

for i e Nq. 

Finally, we define the process X as follows: for a measurable B c 5^, 



if = X, 3 i : T; ^ t < o",-, 
if Xt = X, 3 i : o"; ^ t < T;+i, 



P(X,+ieB|^,)=P(7ieB|7o = x), 



_ P e B I J?,) = P (Z,+i_<,^ e B I Z,_,^ = x) . 

To wit, the process defined above behaves as the homogeneous chain 7 whenever 
it is inside K. Once the process X exits the set K, a controller alters the behavior of the 
chain which, until it enters K again, behaves as a copy of the inhomogeneous chain 
Z starting from a point outside K. The process X is in general non-Markovian due to 
the possible time inhomogeneity of Z. Nevertheless, it is a natural class of examples of 
switching systems whose Markovian behavior switches in different regions on the state 
space. We say thatZ is (7, Z)-hybrid with respect to K. 

The following generalization of Theorem (2.2) can be proved along lines of the 
original proof. The only requirement is a slight modification of the condition (2.3) 
which is needed to alter the second inequality in (2.7). 

(2.9) . Theorem. Consider a stochastic process X that is (Y,Z)-hybrid with respect to 
a measurable K cz 5^ for some homogeneous Marliov chain Y and some possibly in- 
homogeneous Markov chain Z. Suppose Assumption (2.1) holds for the process Z 
and 

(2.10) 13 := sup E [(/'(0,7i)l(y,e^\jc} | = Jo] < oo. 

// the process X starts from Xq& 5^ \K, we have 

(2.11) supE,^[nx,)] ^CI3 + 5 + r^(_0,Xo). 

It is interesting to note that the right side of above bound is a total of individual con- 
tributions by the control (for C), the choice oi K (for 5), and the initial configuration 
(for Xq) . We stress that the conclusion holds even when X is no longer a Markov chain due 
to the time inhomogeneity of Z. This is important, especially because operator- theoretic 
bounds like Foster-Lyapunov, or martingale-based bounds do not work in such a case. 

2.3. Connection with Optimal Stopping Problems. Suppose that we are given a 
Markov chain Z taking values in 5^, a function V : 5^ — > R, and a measurable target 
or safe set K c 5^. (Alternatively, we may assume that we are given an 5^ -valued 
process X that is (7,Z)-hybrid with respect to a measurable K cz 5^.) Our objective is 
to investigate whether the sequence (^(^t))teNo i-i-bounded. To this end one can 
follow the two-step procedure of first searching for a function ip satisfying Assumption 
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(2.1), followed by an application of Theorem (2.9). A systematic procedure of doing 
this is given by the following connection with Optimal Stopping problems. 

Let (0(t))t€No some positive sequence of numbers such that XteNo ^^^^ finite. 
Define the pay-ojf or the reward function as 



V(x)/0(t) if X e 5^ \K, t e Nq, 
ifx eJC, t sNq. 



Recall that the Optimal Stopping problem [PS06, Chapter 1] for the process Z and 
the reward function h defined above consists of finding a stopping time t* such that 

(2.12) E^[h(T*ATj^,Z^.^^j] =esssupE^[h(T At^.Z^^^J], 

where t^^ is the hitting time to the set K, and ess sup refers to essential supremum over 
the set of all possible stopping times (see [PS06, Chapter 1, Lemma 1.3]). 
Define the value function as 

(2.13) (/7(n, Xo) := ess sup E [h(T, V(Z^)) I Z„ = Xq] , 

T6T„ 

where T„ is the set of stopping times 

{(t V n) a I t an arbitrary stopping time}. 

(2.14) . Theorem. Suppose that the value function (/^(O, Xq) is finite for all Xq e 5^, 
then 

(i) ( f , Xq) is finite for all t e Nq and 

^it,Xo) > V(xo)/0(t) fora77(t,xo) eNo x i^\K). 

(ii) The process (Y J ^^j^^ defined by 

Yt ■.= ip{tAz^,Z,^^J 

is a supermartingale. 

Proof. The proof follows from the general theory of optimal stopping. See, for example, 
[CRS71, Chapter 4]. The sequence of rewards is given by the process V(Zt^,. )/0(t), 
t = 0, 1, 2, . . .. Applying [CRS71, Theorem 4.1, p. 66] we get 

t^(n,Xo)= (V(xo)/0(n)) V (^E [(/^(n + l,Z(„+i)^^^) Z„^^^ =Xo]). 

By considering the first of the two terms in the maximum on the right-hand side above 
we obtain (i), and (ii) follows from the second. □ 

In other words, the value function (/'(t,x) defined in (2.13) satisfies the conditions 
of Theorem (2.9). 

(2.15) . Theorem. Consider an 5^ -valued process X that is (Y,Z)-hybrid with respect 
to a measurable K cz 5^ as in §2.2. Suppose that for some nonnegative integrable 
sequence (0(t))tgN„ optimal stopping problem (2.12) has a finite value function 
(/3(t,Xo). If additionally condition (2.10) is true, then the bound (2.11) holds. 

Let us remark that the value function, being the envelope, is the smallest super- 
martingale (hence the sharpest bound) that can satisfy Theorem (2.9). Several meth- 
ods of solving optimal stopping problems in the Markovian setting are available and 
we refer the reader to [PS06] for a complete review. 
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2.4. A Class of Sampled Diffusions. In the setting of the process X being (,Y,Z)- 
hybrid with respect to a given set K, suppose that the state-space for the Markov chains 
Y and Z is K'' and the safe set K is compact. Observe that the only challenge in 
applying Theorem (2.9) is to find a suitable function if given the Markov chain Z and 
the function V. In applications, a natural choice for the function V is given by square 
of the Euclidean norm, i.e., V{x) = XifLi • this choice of V, we describe below 
a natural class of examples of Markov chains for which one can construct a c/j that 
satisfies part (i) of Assumption (2.1). 

Consider a diffusion with a possibly time-inhomogeneous drift function, given by 
the d -dimensional stochastic differential equation 

(2.16) (iX, = bit,Xjdt + dWt, 

where = (W^^l), Wt(2), . . . , Wt(d)) is a vector of d independent Brownian motions, 
and b : R^q x R'^ — > R'* is a measurable function. 

We will abuse the notations somewhat and construct a function if : R^q x R^q — > 
Kj.0 such that [f[t,V[Xi-)))^^^^ is a supermartingale outside a compact set K and 
satisfies f[t,^) ^ ^/0{t) for some nonnegative sequence (^^(t))teNo- define Z; = 
Xif^^ for i e No; Z is the the diffusion sampled at integer time points before hitting 
K. It is clear that Z is a Markov chain such that {fii, ^(■^i)))i<=N„ ^ supermartingale 
that satisfies the Assumptions (2.1) as long as XiteN ^^'-^ ^ 

To construct such a f, let us consider a well known family of one-dimensional 
diffusion, known as the squared Bessel processes (BESQ) . This family is indexed by a 
single nonnegative parameter 5^0 and is described as the unique strong solution of 
the SDE 

(2.17) dY,=2^,db, + 5dt, Yo = yo^O, 

where b := (bt)teNo ^ one-dimensional standard Brownian motion. We have the 
following Lemma: 

(2.18) . Lemma. Let F : R — > R^q be a nonnegative, increasing, and convex function, 
and fix any terminal time S > 0. Define the function 

(2.19) (^(f,j):=E[f(Fs)|n=y], fG[0,S], 

where Y solves the SDE (2.17). Then f satisfies the following properties: 

(i) If is increasing in y , 

(ii) f is convex in y , and 

(iii) f satisfies the partial differential equation 



(2.20) 



-^+5if' + 2yf" = 0, y>0, te(0,S), 
V(.S,y) = Fiy). 



Note that f' and f" in the statement of Lemma (2.17) refers to the first and second 
derivatives with respect to the second argument of f. 



Proof. The proof proceeds by coupling. Let us first show that f is increasing as claimed 
in (i). Fix S > 0. Consider any two starting points ^ x < y. Construct on the same 
sample space two copies of BESQ processes y'^^^ and Y^^^ such that both of them satisfy 
(2.17) with respect to the same Brownian motion b but 7^^' = x and Y^^^ = y. It is 
possible to do this since the SDE (2.17) admits a strong solution (see [KS08, Chapter 
5, Proposition 2.13]). Hence, by [KS08, Chapter 5, Proposition 2.18], it follows that 
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Yf^-' ^ y/^-* for all f > 0. Since F is an increasing function, we get 

This proves that is increasing in the second argument. 

For convexity of ip claimed in (ii) , we use a different coupling. We follow arguments 
very similar to the one used in the proof of [Hob98, Theorem 3.1]. Consider three 
initial points < z < j < x. And let X, Y, Z be three independent BESQ processes that 
start from x, y, and z respectively. Define the stopping times 

T^=inf|u Y^=X^Y T^=inf|u Y^ = Z^y 

Fix a time t e [0,S], and let T = S - t. Define 

O" = Tj. A Tj, A T. 
Now, on the event cj = t^, it follows from symmetry that 

e\{Xt- Zj)F{Yt)^^^,A = E [(Fr - Zt)F{Xt) ^a=.j] , 
(2.21) ^ '- ' 

E[(X^-yr)F(Z^)l{,=,^;] =0. 

Similarly, on the event a = t^, we have 

e\{Xt- ZT)F{Yr) =e\{Xt- Yj)F{Zj) 1;,=,^;] , 

(2.22) 

E[iZr-Yr)FiXr)ha=.j]=0- 

And finally, when cj = T, we must have Zj <Yj- <Xj. We use the convexity property 
of F to get 

£\{Xj-Zj)F{Yj)\,^.rA ^£.\{Xj-Yj)F{Zj)\,^jA 

(2.23) 

+ £[(Yj-Zj)F{Xj)l^,^j^'\. 

Combining the three cases in (2.21), (2.22), and (2.23) we get 

(2.24) E.\{Xj-Zt)F{Yj)\ ^E.\{Xj - Yj)F{Zj)\+E.\{Yj - Zj)F{Xj)\. 

We now use the fact that X, Y, and Z are independent. Also, it is not difficult to see 
from the SDK (2.17) that [Ij-] - x = E^, [Yj] - y = E, [Zj~\ - z = 5t. Thus, from 
(2.24) we infer that 

(x - z)c^(t, j) ^ (x - j)(/j(t,z) + (y - z)c^(t, x), for all < z < y < x. 

This proves convexity of in its second argument. 

Finally, to see (iii), it suffices to observe that the equation (2.20) is the classical 
generator relation for diffusions, for which we refer to [KS08, Chapter 5.4]. The tran- 
sition density of BESQ processes are smooth and have an explicit representation that 
satisfy equation (2.20). The general case can be obtained by differentiating under the 
integral with respect to F. □ 

Let us return to the multidimensional diffusion given by (2.16). We consider the 

process (Ct)teNo' where := ip(t, and ip is the function in (2.19). Note that, 

since F is nonnegative, so is ip. Additionally, since is convex, we have 

ip{t,^)^ip[t,o) + ipXt,o+)^. 

Hence the sequence (^^(t))f^o is given by 

0(f) = l/i/^'(t,O+), f = 0,l,...,S. 
We have the following Theorem: 
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(2.25). Theorem. Suppose that there exists a compact set K c R'* such that that the 
drift function b = [bi,b2,. ■ ■ ,b^) in the SDE (2.16) satisfies the sector condition 

d 

^x,fai(t,x)<0 for it,x)&R;^Qx[S^\K). 

i = l 

Fix any terminal time T > 0. Define the process [(^)^f^m^ := ((^(t, , where 

is the nonnegative, increasing, convex function defined in (2.19) with 

II 1|2 

^(y) = y 5 = d. 

Then, with the set-up as above, the stopped process (CtATjjAr)t>o ^ (local) super- 
martingale. 



Proof. Applying Ito's rule to (Ct)teKjo, we get 

' d If 



(2.26) 



dt 



dt, 



where M := [Mj^^j^^^ is in general a local martingale (M is a martingale under addi- 
tional assumptions of boundedness on the first derivative of (/?), and is the generator 
of X. We compute 



difi 

"a 



2dt^' + t^"^4xf 



+ d^' + 2 U" + 2^'^ b,x, = 2^'2 



where the final equality holds since satisfies (2.20) at j = 2- x\ 

We know that c^' > since is increasing, and, by our assumption, x^b^ < 
whenever x^K. Thus, 



Jt 



+ if(/j^O for (t,x)e [0,T] X (5^ \iiC). 



Now the claim follows from the semimartingale decomposition given in (2.26). 



□ 



Note that the supermartingale (Ct)teNo has been defined only for a bounded tem- 
poral horizon. Thus, to show that Theorem (2.9) holds, some additional uniformity 
assumptions would be needed. 



3. Application to Discrete-Time Randomly Switched Systems 

In this section we look at several cases of discrete-time randomly switched systems 
(or, iterated function systems,) in which Theorem (2.2) of §2 applies and gives useful 
uniform Lj bounds of Lyapunov functions. In §3.1 we give sufficient conditions for 
global asymptotic stability almost surely and in of discrete-time randomly switched 
systems. Assumptions of global contractivity in its standard form or memoryless choice 
of the maps at each iterate are absent; we simply require a condition resembling av- 
erage contractivity in terms of Lyapunov functions with a suitable coupling condition 
with the Markovian transition probabilities. In §3.2 we demonstrate that under mild 
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hypotheses iterated function systems possess strong stability and robustness properties 
with respect to bounded disturbances that are not modelled as random processes.^ 

3.1. Stability of Discrete-Time Randomly Switched Systems. Consider the system 

(3.1) x,+i=/,/X,), Xo = Xo, teNo. 

Here a : Nq — > ^ := {1,...,N} is a discrete-time random process, the map /; : 
K'' — > K'' is continuous and locally Lipschitz, and there are points x* e R'' such that 
= for each i e The initial condition of the system Xq e M.'^ is assumed to 
be known. Our objective is to study stability properties of this system by extracting 
certain nonnegative supermartingales. 

The system (3.1) can be viewed as an iterated function system: X^^i ~ fcr, ° ■■■ ° 
fa^ °/(T(,(^o)- Varying the point Xq but keeping the same maps leads to a family of 
Markov chains initialized from different initial conditions. The article [DF99] treats ba- 
sic results on convergence and stationarity properties of such systems with the process 
(o't)t€No being a sequence of independent and identically distributed random variables 
taking values in 0^ , and each map /, is a contraction. These results were generalized 
in [JTOl] with the aid of Foster-Lyapunov arguments. 

The analysis carried out in [JTOl] requires a Polish state-space, and employs the 
following three principal assumptions: (a) the maps are non-separating on an aver- 
age, i.e., the average separation of the Markov chains initialized at different points is 
nondecreasing over time; (b) there exists a set C such that the Markov chains started 
at different initial conditions contract after the set C is reached; and (c) there ex- 
ists a measurable real-valued function V ^ 1, bounded on C, and satisfying a Foster- 
Lyapunov drift condition QV(x) ^ XV{x) + blc(x) for some A e ]0, 1[ and h < oo, 
where Q is the transition kernel. Under these conditions the authors establish the ex- 
istence and uniqueness of an invariant measure which is also globally attractive, and 
the convergence to this measure is exponential. In particular, this showed that the 
main results of [DF99], which are primarily related to existence and uniqueness of 
invariant probability measures, continue to hold if the contractivity hypotheses on the 
family {/Jie^ are relaxed. In this subsection we look at stronger properties, namely, 

boundedness and stability, and almost sure stability of the system (3.1) under As- 
sumption (2.1). No contractivity inside a compact set is needed to establish existence 
of an invariant measure under Assumption (2.1). 

(3.2) . Assumption. The process {ut)t&ia irreducible Markov chain with initial 
probability distribution n° and a transition matrix P := [pfjlNxN- ^ 

It is immediately clear that the discrete-time process (cTcXJtgjj^, taking values in 
the Borel space ^ x K'', is Markovian under Assumption (3.2). The corresponding 
transition kernel is given by 

Q((i,jc),^' xB) =Xijg<j,/Pi;lB(/jU)) for ^'c^,B a Borel subset of R'*, 

and (i,x) x R'*. 

Our basic analysis tool is a family of Lyapunov functions, one for each subsystem, 
and at different times we shall impose the following two distinct sets of hypotheses on 
them.^ 

^Recall the following notation: We let Jl^ denote the collection of strictly increasing continuous func- 
tions a : ffijo — * such that a(0) = 0; we say that a function a belongs to class-J^Too if a 6 J^f and 
hnir^oo a(r) = oo. A function /3 : R^g ^ f^o — > Kj.o belongs to dass-J(^S^ if I3{-, n) e JX' for a fixed n e Nq, 
and if /3(r, n) ^ as n ^ cxd for fixed r e K^g. Recall that a function / : R"^ — > R"* is locally Lipschitz 
continuous if for every Xg e R** and open set O containing Xg, there exists a constant i > such that 
||/(^) -/(^o)|| ^ i ||x - Xo|| whenever x e O. 

■^It will be useful to recall here that the deterministic system Xf+j = /i(Xf), t e Nq, with initial condition 
Xq is said to be globally asymptotically stable (in the sense of Lyapunov) if (a) for every £ > there exists a 
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(3.3) . Assumption. There exist a family {V;};gg!, of nonnegative measurable functions 
on R'*, functions a^, ^ numbers e ]0, 1[, r > and /i > 1, such that 

(Vl) ai(||x — ^ V,(x) ^ HjCjl^ ~ ^*||) for all X and I, 
(V2) V,(x) ^ ptVj(x) whenever ||x II > r, for all and 

(V3) V;(/,(x)) ^ A„V,(x) for all X and i. 

(3.4) . Assumption. There exist a family {Vj^gga of nonnegative measurable functions 
on R'', functions a^, 02 e JiT, a matrix [A^j J^xn with nonnegative entries, and numbers 
r > 0, /i > 1, such that (V1)-(V2) of Assumption (3.3) hold, and 

(V3') Vi(/j(x))^ A,j V,(x) forallx and 

The condition (VI) in Assumption (3.3) is standard in deterministic system theory 
literature, ensuring, in particular, positive definiteness of each . CV2) stipulates that 
outside the functions {Vj,gga are linearly comparable to each other The conditions 
(VI) and CV3) together imply that each subsystem is globally asymptotically stable, 
with sufficient stability margin — the smaller the number A^, the greater is the stability 
margin. In fact, standard converse Lyapunov theorems show that (Vl) and CV3) are 
necessary and sufficient conditions for each subsystem to be globally asymptotically 
stable. The only difference between Assumptions (3.3) and (3.4) is that the latter keeps 
track of how each Lyapunov function evolves along trajectories of every subsystem. 

Let us define p := maxp,, and p := max p, . 

(3.5) . Proposition. Consider the system (3.1), and suppose that either of the following 
two conditions holds: 

(51) Assumptions (3.2) and (3.3) hold, and A^Cp < 1. 

(52) Assumptions (3.2) and (3.4) hold, and fx- {jnax^^^'^jeff' Pij^ji^ ^ ^■ 

LetT^ := inf{t e Nq] \\Xt\\ ^ r} and V.'[x) := V;(x)li[j,(\g (x). Suppose that ||xo|| > r. 
Then there exists a> such that the process (e"'^'^^' (^tAT,.))t€No ^ nonnega- 
tive supermartingale. 

(3.6) . Corollary. Consider the system (3.1), and assume that the hypotheses of Propo- 
sition (3.5) hold. Then there exists a constant c > such that sup Erai( XJ )] < c. 

t6N„ 

It is possible to derive simple conditions for stability of the system (3.1) from Propo- 
sition (3.5). To this end we briefly recall two standard stability concepts. 

(3.7) . Definition. If ker(/, - id) = {0} for each i e ^, the system (3.1) is said to be 
o globally asymptotically stable almost surely if 

(ASl) P(V£:>0 35 >0 s.t. sup llxJI < e whenever ||xo|| < 5 ) =1, 

(AS2) PrVr,e'>0 3T >Os.t. sup llxJI < e' whenever ||xo|| < = 1; 

o a-stable in Ij for some a e JiT if 

(SMI) Ve> 35 >0s.t. sup E ra(||xJhl < e whenever ||xo|| < 5, 

t€No 

(SM2) V r, e' > 3 T > s.t. sup E ra(||xJhl < e' whenever ||xo|| < r. 

(3.8) . Corollary. Suppose that ker(/; — id) = {0} for each i e and that either of the 
hypotheses (SI) and (S2) of Proposition (3.5) holds with r = 0. Then 

5 > such that ||xo - x*|| < 5 imphes ||xt - x*|| < s for all t e Nq, and (b) for every r,s' > there exists 
a r > such that ||xo - x*|| < r implies |jxt - x*|| < e for all t > T. The condition (a) goes by the name 
of Lyapunov stability of the dynamical system (or of the corresponding equilibrium point x*), and (b) is the 
standard notion of global asymptotic convergence to x* . 
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o there exists a> such thatlinif^^ E\_e'^^V^ (^t)] = 0' ^^'^ 
o the system (3.1) is globally asymptotically stable almost surely and a^-stable in 
in the sense of Definition (3.7). 

The proofs of Proposition (3.5), Corollary (3.6) and Corollary (3.8) are given after 
the following simple Lemma; the crude estimate asserted in it resembles the distribu- 
tion of a Binomial random variable, except that we have p + p ^ 1. For t e N let the 
random variable denote the number of times the state of the Markov chain changes 
on the period of length t starting from 0, i.e., JV^ := Xii=i llan^CTi}- 

(3.9) . Lemma. Under Assumption (3.2) we have fors < t, s, t e Nq, 

else. 

Proof. Fix s < t,s,t & Nq, and let tj^(s, t) := P(JVf - N, = k\a^). Then by the Markov 
property, for fc = 0, 1, . . . , t — s, 

rj,(s, t) = rj,(s, t - 1)P(JV, - JV, = fc|jV,_i - JV, = fc, a,) 

+ rj,_i(5, t - 1)P(JV, - JV, = fc|jV,_i - JV, = fc - 1, C7,) 

^ Plkis, t - 1) +P')7fc-l(5, t - 1). 

The set of initial conditions 17,(5, t) = for all i 5^ t — s, follow from the trivial ob- 
servation that there cannot be more than t — s changes of cr on a period of length 
t — s. This gives a well-defined set of recursive equations, and a standard induction 
argument shows that 17^(5, t) ^ Ck^)P'^'^~'^~'^^P'^- '^^^^ proves the assertion. □ 

Proof of Proposition (3.5). First we look at the assertion under the condition (SI). Fix 

s < t, 5, t e No. Given (ct,^^ from (V3) we get v;^^^^ (X^s+dat,.) ^ ^X,,,^_ (^sat,), 

and if ct.+i # cj„ we employ (V2) to get V^^^^^^^^ (X^.+d^^ ) ^ mV^^^^ (^(5+i)at,)- 
Therefore, 

^k+mrr (^('^+i)AT,-) ^ /^^"^a,,,,. (^.AT,,) if o-(,+i)AT, 7^ CTsAz,, and 
v;^^^^^^^ (^(.+i)Ax,) ^ otherwise. 

Iterating this procedure we arrive at the pathwise inequality 

(3.10) v;^^^ ^ v;^^^ 

Since s A t,. = t A 5 A t,-, and t A t,. is measurable with respect to Scasat > we invoke 
the Markov property of [a^,XJi-^f^^to arrive at 

E[K^„,.(^tAx,)|KAT,,^.AT,)] 

We now apply the estimate in Lemma (3.9) to get E |^ju'^"--' (CTj^^ ,Xsat- )j ^ 
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Since Ao(p + jup) < 1, letting a! := A^Cp + /^p)e" < 1, the above inequality gives 
(3.11) L ' ' 'J 

This shows that (e"'^'^'^' (-^tAx, ) ) t€No ^ nonnegative supermartingale. 

Let us now look at the assertion of the Proposition under the condition (S2). Fix t e 
No- Then from (V30, V^'(/a„,,^ (^tAT,)) ^ V^' (^tAx J for all ; e and by (V2), 

This leads to 

E[^4+i)Ar,-*^^{f+i>^.^ (f^tAx,'^tAx,)] max^p,j-Aj-, V^^^^ (X^^^J. 

Since by hypothesis there exists a > such that ^max^gga X^gg^ Pij-^ji) < 1, the 
last inequality shows immediately that (^e'^'^'^^' V^ (-^tAx,))teNo ^ supermartin- 
gale. This concludes the proof □ 

Proof of Corollary (3.6). First observe that since each map /; is locally Lipschitz, the 
diameter of the set D, := e B^} is finite, and since ^ is finite, so is the 

diameter of IJ^ggj -D;. Therefore, if Q is the transition kernel of the Markov process 
(CTt,Xf then emplojang (Vl) and the fact that /; is locally Lipschitz for each i, we 
arrive at 



^P.;1e''\B,(/;Uo))«2(||/;Uo)||) ^ ^P.j^ ||^o|| < Lr < 00 



for ||xq|| < r, where L is such that supj^^^^g^ ^ L \\y\\- This shows that 

condition (2.3) of Theorem (2.2) holds under our hypotheses, and by Proposition (3.5) 

we know that there exists a> such that ^e"'^'^^'^' V^.^^ (X^^^ ) l^d^g^ (Xj^^. ) j is 

a supermartingale. Theorem (2.2) now guarantees the existence of a constant C' > 
such that supfgj^^ E [Vo.^(Xt)l]jdy^ (Xjj ^ C', and finally, from (Vl) it follows that 
there exists a constant c > such that supjgpj^ E[ai(||Xt||)j ^ c < oo, as asserted. □ 

Proof of Corollary (3.8). We prove almost sure global asymptotic stability and sta- 
bility in Lj of (3.1) under the condition (SI) of Proposition (3.5); the proofs under 
(S2) are similar 

First observe that since ker(/; — id) = {0} for each i e i.e., is the equilibrium 
point of each individual subsystem, Px„i'^{o} < oo) = for Xq ^ 0, where Tjoj is the 
first time that the process (Xj^gj^^ hits {0}. Indeed, since ker(/; — id) = {0} for each 
i e 5» and # we have Q((i,Xo),^ x {0}) = Xijgg.p,jl{o}(/j(-^o)) = 0- which 
shows that Q"((i,Xo),^ x {0}) = whenever Xq ^ 0. The observation now follows 
from P^„(t{o} < oo) = P;,„ (U„sn{'^{o} = '^D ^ Ii„eN Px„ (tjo} = n). Therefore, 
with T|o} = Tr = proceeding as in the proof of Proposition (3.5) above, one can 
show that (e"'^V„^ (Xt))jgj^ is a supermartingale for some a > 0. In particular. With 
5 = and = 00 in (3.11), we apply (VI) to arrive at lim^..^^ E [e'^'^V^ (^t)] = 
limt^^E[E[e«fV<,^(Xj|(cTo,Xo)]] ^ lim^^,^ a2(||xo||)(aO' = 0. Standard super- 
martingale convergence results and the definition of Tjo( imply that P^limt_oo V^^iX^) = 
= 1. With 5 = and = TjQj = oo, the pathwise inequality (3.10) in conjunction 
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with (VI) give Vf^^iX^) ^ a2(||-^o||)M'^'-^l- The foregoing inequahty impUes that for 
almost every sample path (o"f,Xptgf^^ corresponding to initial condition = x'q with 
XqII < ||^o||) one has 

lim ^ lim a2(||x^||)M^'A^ ^ lima2i\\xo\\)n'''K = 0, 

which proves (AS2). Since the family {/Jiega is finite, and each /; is locally Lipschitz, 
there exists L > such that sup,g^ ^ L \\x\\ whenever ^ 1. Fix e > 0. By 

(AS2) we know that for almost all sample paths there exists a constant T > such that 
supjjjj- < E whenever ||xo|| < 1. Then the choice of 5 = (eL~^) A 1 immediately 
gives us the (ASl) property. 

It remains to verify (SMI) and (SM2). Both the properties follow from (3.11) in the 
proof of Proposition (3.5), with s = and = Tjo( = 0. Indeed, with these values of 
s and T^, (3.11) becomes 

E[e-'a,i\\x,\\)\[aM] ^ E[e«^V,/Xj|(c7o,Xo)] ^ V,^iX,XaJ ^ a2(||xo||)(a')^ 

in view of (VI), where a' = X„{p + fJ,p)e'^ < 1. Therefore, given e > 0, we sim- 
ply choose 5 < a2^[e) to get (SMI). Given r, e' > 0, we simply choose T = V 
(In(a2(r)/e0/ln(a')) to get (SM2). This completes the proof □ 

3.2. Robust Stability of Discrete-Time Randomly Switched Systems. Conditions 
for the existence of the supermartingale [e"'-^'^'^>^^V(X[^^^))^^^^ in §2 can be easily 
expressed in terms of the transition kernel Q. However, if Q is not known exactly, which 
may happen if the model of the underlying system generating the Markov process 
(-^t)t€No uncertain, one needs different methods. We look at one such instance 
below. 

Consider the system 

(3.12) ^t+i=/a,(^t,wj, Xo=Xo, fGNo, 

where we retain the definition a from §3.1, /; : K'' x K'" — > M.'^ is locally Lipschitz 
continuous in both arguments with /,(0, 0) = for each i e 0^, and (wj^gj^^ is a 
bounded and measurable R'" -valued disturbance sequence. We do not model (wJtgpj^ 
as a random process; as such, the transition kernel of (3.12) is not unique. 

(3.13) . Definition. The system (3.12) is said to be input-to-state stable in Ij if there 
exist functions x,x' ^ -^oo and ip e JT^ such that E^,^ [l(||^t||)] ^ ''/'(||-^o|| . + 
sup,eNoZ'(|h.||)forall teNo- 

Our motivation for this definition comes from the concept of input-to-state stability 
iss in the deterministic context [JWOl]. Consider the i-th subsystem of (3.12) x^^i = 
/,(Xf,M/t) for t e No with initial condition Xq; note that (xJtgj^^ is a deterministic 
sequence. This nonlinear discrete-time system is said to be iss if there exist functions 

e JTif and z e JT^ such that ||xf|| ^ iZ-CUxoH , t) sup^^jj^ for t e Nq. A 

sufficient set of conditions (cf [JWOl, Lemma 3.5]) for iss of this system is that there 
exist a continuous function V : K'' — > Mj,o, aj, e J^^q, p e J^, and a constant A e 
]0, 1[, such that ai(||x||) ^ V(x) ^ ct2(ll^ll) for all x e R'^, and V(/i(x,w)) ^ AV(x) 
whenever ||x|| > p(||w||). 

In this framework we have the following Proposition. 

(3.14) . Proposition. Consider the system (3.12), and suppose that 

(i) Assumption (3.2) holds, 

(ii) there exist continuous functions V; : R'' — > R^q for i e 0^, ai,a2,p e ^ooj ^ 
constant IX > 1 and a matrix [A^j J^xn of nonnegative entries, such that 

(a) ai(||x||)^V,.(x)^a2(lkll) for all x and i, 

(b) Vi{x) ^ mV,(x) for all x and i,j, and 
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(c) Viifjix)) ^ A,j Vi(x) whenever \\x\\ > p(||w||) and all 
(iii) M(max,g<5,Xijg£y Pi;'^j.) < 1- 

Then (3.12) is input-to-state stable in Lj in the sense of Definition (3.13). 

Proof. We define the compact set J<C := e ^ ^ sup^^j,^ p(||w5||)}, and 

let := inf{ t e No|Xt e K}. In this setting we know from the preceding analysis that 
i/7(t,^) = e"'^, 0[t) = e""', and C = 1/(1 - e""). We see from the estimate (2.8) in 
the proof of Theorem (2.2) that 

Ex„ [VaS^J] ^ ^{0, V<,„(xo))0(f) + + 5 ^ a2(||xo||)e-«^ + + 5. 

Standard arguments show that there exists some e such that ^ and 5 are each 
dominated by x"(suPseNo ||^s||)» therefore, there exists some x' ^ -^oo such that 
^/(l — e~") + 5 is dominated by /'(sup^^pj^ ll^sll)- -^PPlyii^S (ii)(a) on the left-hand 
side of the last inequality, we conclude that (3.12) is input-to-state stable with / = cti 
and '(/)(r, t) = a2('")e"°^ □ 
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